Content Agnostic Malware Detection in Networks

Bots are the root cause of many security problems on the Internet – they send spam, steal information from infected machines, and perform distributed denial of service attacks. Given their security impact, it is not surprising that a large number of techniques have been proposed that aim to detect and mitigate bots, both network-based and host-based approaches. Detecting bots at the network-level has a number of advantages over hostbased solutions, as it allows for the efficient analysis of a large number of hosts without the need for any end point installation. Currently, network-based botnet detection techniques often target the command and control traffic between the bots and their botmaster. Moreover, a significant majority of these techniques are based on the analysis of packet payloads. The proposed approaches range from simple pattern matching against signatures to structural analysis of command and control communication. Unfortunately, deep packet inspection is rendered increasingly ineffective as malware authors start to use obfuscated or encrypted command and control connections. This thesis presents BotFinder, a novel system that can detect individual, malware-infected hosts in a network, based solely on the statistical patterns of the network traffic they generate, without relying on content analysis. BotFinder uses machine learning techniques to identify the key features of command and control communications, based on observing traffic that bots produce in a controlled environment. Using these features, BotFinder creates models that can be deployed at edge routers to identify infected hosts. The system was trained on several different bot families and evaluated on real-world traffic datasets – most notably, the NetFlow information of a large ISP that contains more than 25 billion flows, which correspond to approximately half a Petabyte of network traffic. The results show that BotFinder achieves high detection rates with very low false positives.